{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Model Evaluation\n",
        "\n",
        "In this notebook you will learn:\n",
        "\n",
        "- What is model evaluation?\n",
        "- How to evaluate a model in ML.NET\n",
        "- Train and evaluate models with cross-validation\n",
        "- Model explainability\n",
        "- How to improve your model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## What is model evaluation?\n",
        "\n",
        "Training is the process of applying algorithms to historical data in order to create a model that accurately represents that data. That model is then used on new data to make predictions. \n",
        "\n",
        "Model evaluation is the process of using metrics to quantify how effectively your model learned patterns within your data and applies those learnings to new and unseen data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## How to evaluate a model in ML.NET"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Reference ML.NET Daily Build Feed"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div><strong>Restore sources</strong><ul><li><span>https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json</span></li></ul></div><div></div><div></div></div>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#i \"nuget:https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div><strong>Restore sources</strong><ul><li><span>https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json</span></li></ul></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.Data.Analysis, 0.20.0-preview.22356.1</span></li><li><span>Microsoft.ML.AutoML, 0.20.0-preview.22356.1</span></li></ul></div></div>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "text/markdown": [
              "Loading extensions from `Microsoft.Data.Analysis.Interactive.dll`"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "text/markdown": [
              "Loading extensions from `Microsoft.ML.AutoML.Interactive.dll`"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#r \"nuget: Microsoft.ML.AutoML, 0.20.0-preview.22356.1\"\n",
        "#r \"nuget: Microsoft.Data.Analysis, 0.20.0-preview.22356.1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Reference packages with `using` statements"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using System.Text.Json;\n",
        "using Microsoft.Data.Analysis;\n",
        "using Microsoft.ML;\n",
        "using Microsoft.ML.AutoML;\n",
        "using Microsoft.ML.Data;\n",
        "using Microsoft.ML.Trainers.FastTree;\n",
        "using static Microsoft.ML.Transforms.OneHotEncodingEstimator;"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Download or Locate Data\n",
        "The following code tries to locate the data file in a few known locations or it will download it from the known GitHub location."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Train Data Path: C:\\dev\\csharp-notebooks\\machine-learning\\data\\taxi-fare.csv\r\n"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "using System;\n",
        "using System.IO;\n",
        "using System.Net;\n",
        "\n",
        "string EnsureDataSetDownloaded(string fileName)\n",
        "{\n",
        "\n",
        "\t// This is the path if the repo has been checked out.\n",
        "\tvar filePath = Path.Combine(Directory.GetCurrentDirectory(),\"data\", fileName);\n",
        "\n",
        "\tif (!File.Exists(filePath))\n",
        "\t{\n",
        "\t\t// This is the path if the file has already been downloaded.\n",
        "\t\tfilePath = Path.Combine(Directory.GetCurrentDirectory(), fileName);\n",
        "\t}\n",
        "\n",
        "\tif (!File.Exists(filePath))\n",
        "\t{\n",
        "\t\tusing (var client = new WebClient())\n",
        "\t\t{\n",
        "\t\t\tclient.DownloadFile($\"https://raw.githubusercontent.com/dotnet/csharp-notebooks/main/machine-learning/data/{fileName}\", filePath);\n",
        "\t\t}\n",
        "\t\tConsole.WriteLine($\"Downloaded {fileName}  to : {filePath}\");\n",
        "\t}\n",
        "\telse\n",
        "\t{\n",
        "\t\tConsole.WriteLine($\"{fileName} found here: {filePath}\");\n",
        "\t}\n",
        "\n",
        "\treturn filePath;\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var trainDataPath = EnsureDataSetDownloaded(\"taxi-fare.csv\");\n",
        "var df = DataFrame.LoadCsv(trainDataPath);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Once the data is loaded, use the `Head` method to preview the first five rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table id=\"table_637934937843853168\"><thead><tr><th><i>index</i></th><th>vendor_id</th><th>rate_code</th><th>passenger_count</th><th>trip_time_in_secs</th><th>trip_distance</th><th>payment_type</th><th>fare_amount</th></tr></thead><tbody><tr><td><i><div class=\"dni-plaintext\">0</div></i></td><td>CMT</td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1271</div></td><td><div class=\"dni-plaintext\">3.8</div></td><td>CRD</td><td><div class=\"dni-plaintext\">17.5</div></td></tr><tr><td><i><div class=\"dni-plaintext\">1</div></i></td><td>CMT</td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">474</div></td><td><div class=\"dni-plaintext\">1.5</div></td><td>CRD</td><td><div class=\"dni-plaintext\">8</div></td></tr><tr><td><i><div class=\"dni-plaintext\">2</div></i></td><td>CMT</td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">637</div></td><td><div class=\"dni-plaintext\">1.4</div></td><td>CRD</td><td><div class=\"dni-plaintext\">8.5</div></td></tr><tr><td><i><div class=\"dni-plaintext\">3</div></i></td><td>CMT</td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">181</div></td><td><div class=\"dni-plaintext\">0.6</div></td><td>CSH</td><td><div class=\"dni-plaintext\">4.5</div></td></tr><tr><td><i><div class=\"dni-plaintext\">4</div></i></td><td>CMT</td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">661</div></td><td><div class=\"dni-plaintext\">1.1</div></td><td>CRD</td><td><div class=\"dni-plaintext\">8.5</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.Head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Initialize MLContext\n",
        "\n",
        "All ML.NET operations start in the [MLContext](https://docs.microsoft.com/dotnet/api/microsoft.ml.mlcontext) class. Initializing mlContext creates a new ML.NET environment that can be shared across the model creation workflow objects. It's similar, conceptually, to DBContext in Entity Framework."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var mlContext = new MLContext();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Split the data into train, validation and test sets\n",
        "\n",
        "The original dataset is split into three subsets: train, validation, and test. The **train** set is what you'll use to learn the patterns of your data. The **validation** set is used to optimize the model hyperparameters. The **test** set is used to evaluate the performance of your model using evaluation metrics for the regression task. \n",
        "\n",
        "In this case, 80% of the data is used for training as defined by the `testFraction` parameter. The remaining 20% is split in half and used as the validation and test sets. \n",
        "\n",
        "#### Why not use the entire dataset?\n",
        "\n",
        "Although in general providing your trainer with more examples it can learn from is recommended, you don't want a model that only performs well on historical data. Instead, you're looking for a model that can learn from that historical data and generalize or make accurate predictions on new and unseen data. \n",
        "\n",
        "Some common problems you encounter during training are overfitting and underfitting. Underfitting means the selected trainer is not capable enough to fit training dataset and usually results in a high loss during training and low score/metric on test dataset. To resolve this you need to either select a more powerful model or perform more feature engineering. Overfitting is the opposite, which happens when the model learns the training data too well. This usually results in low loss metric during training but high loss on test dataset.\n",
        "\n",
        "A good analogy for these concepts is studying for an exam. Let's say you knew the questions and answers ahead of time. After studying, you take the test and get a perfect score. Great news! However, when you're given the exam again with the questions rearranged and with slightly different wording you get a lower score. That suggests you memorized the answers and didn't actually learn the concepts you were being tested on. This is an example of overfitting. Underfitting is the opposite where the study materials you were given don't accurately represent what you're evaluated on for the exam. As a result, you resort to guessing the answers since you don't have enough knowledge to answer correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var trainTestData = mlContext.Data.TrainTestSplit(df,testFraction:0.2);\n",
        "var validationTestData = mlContext.Data.TrainTestSplit(trainTestData.TestSet,testFraction:0.5);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var trainSet = trainTestData.TrainSet;\n",
        "var validationSet = validationTestData.TrainSet;\n",
        "var testSet = validationTestData.TestSet;"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create training pipeline\n",
        "\n",
        "For this dataset, the following transforms are applied:\n",
        "\n",
        "- `OneHotEncoding` to convert categorical values into numerical values\n",
        "- `ReplaceMissingValues` which as the name suggests is to replace any missing values.\n",
        "- `Concatenate` takes all of the features and creates a feature vector\n",
        "\n",
        "AutoML is used to define a regression experiment using the `fare_amount` column as the column to predict or label column. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pipeline = \n",
        "\tmlContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair(@\"vendor_id\", @\"vendor_id\"), new InputOutputColumnPair(@\"payment_type\", @\"payment_type\")},outputKind: OutputKind.Binary)\n",
        "\t\t.Append(mlContext.Transforms.ReplaceMissingValues(new[] { new InputOutputColumnPair(@\"rate_code\", @\"rate_code\"), new InputOutputColumnPair(@\"passenger_count\", @\"passenger_count\"), new InputOutputColumnPair(@\"trip_time_in_secs\", @\"trip_time_in_secs\"), new InputOutputColumnPair(@\"trip_distance\", @\"trip_distance\") }))\n",
        "        .Append(mlContext.Transforms.Concatenate(@\"Features\", new[] { @\"vendor_id\", @\"payment_type\", @\"rate_code\", @\"passenger_count\", @\"trip_time_in_secs\", @\"trip_distance\" }))\n",
        "        .Append(mlContext.Auto().Regression(labelColumnName: \"fare_amount\"));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Configure experiment\n",
        "\n",
        "Use AutoML to configure our experiment to train for 60 seconds using the pipeline you've just defined. \n",
        "\n",
        "By default, AutoML evaluates the models it trains using the evaluation metric you want to optimize. In this case it's R-Squared which is calculated by comparing the actual value `fare_amount` against the predicted value `Score`. \n",
        "\n",
        "Evaluation metrics are highly dependent on the task. For regression, some common metrics include:\n",
        "\n",
        "- Mean Absolute Error (MAE)\n",
        "- Mean Squared Error (MSE)\n",
        "- Root Mean Squared Error (RMSE)\n",
        "- R-Squared\n",
        "\n",
        "Your dataset and what you're trying to achieve highly impact your metric selection. If you have outiers in your dataset, it may skew your predictions. MAE, MSE, and RMSE calculate the distance between the predicted and actual data points. All of these measures are sensitive to outliers so if you have outliers in your dataset, they will show up in your metrics. R-Squared calculates the correlation between the actual and predicted values. However, as you add more data points, your R-Squared may continue to increase giving the false impression that a model with high a high R-Squared value has good predictive capabilities. The result of a high R-Squared value can someitmes indicate overfitting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var experiment = \n",
        "\tmlContext.Auto().CreateExperiment()\n",
        "\t\t.SetPipeline(pipeline)\n",
        "        .SetTrainingTimeInSeconds(60)\n",
        "        .SetDataset(trainSet, validationSet)\n",
        "        .SetEvaluateMetric(RegressionMetric.RSquared, \"fare_amount\", \"Score\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Run experiment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var result = await experiment.RunAsync();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### View evaluation metrics of best model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "R-Squared: 0.9329530384286037"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "$\"R-Squared: {result.Metric}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Note that during training, the evaluation metrics were calculated using the validation set. To see how well your model performs on new data, evaluate its performance against the test set. \n",
        "\n",
        "Start by getting the best model using the `Model` property from the training results. Then, use the `Transform` method to use the model to make predictions against the test dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "ITransformer bestModel = result.Model;\n",
        "var predictions = bestModel.Transform(testSet);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Inspect the first few predictions (`Score` column) and compare them against the actual value (`fare_amount` column). Then, calculate the difference between them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th><i>index</i></th><th>Actual</th><th>Predicted</th><th>Difference</th></tr></thead><tbody><tr><td>0</td><td><div class=\"dni-plaintext\">24.5</div></td><td><div class=\"dni-plaintext\">23.087162</div></td><td><div class=\"dni-plaintext\">1.412838</div></td></tr><tr><td>1</td><td><div class=\"dni-plaintext\">9.5</div></td><td><div class=\"dni-plaintext\">8.993666</div></td><td><div class=\"dni-plaintext\">0.5063343</div></td></tr><tr><td>2</td><td><div class=\"dni-plaintext\">4.5</div></td><td><div class=\"dni-plaintext\">4.808011</div></td><td><div class=\"dni-plaintext\">-0.30801105</div></td></tr><tr><td>3</td><td><div class=\"dni-plaintext\">8</div></td><td><div class=\"dni-plaintext\">7.994398</div></td><td><div class=\"dni-plaintext\">0.005601883</div></td></tr><tr><td>4</td><td><div class=\"dni-plaintext\">52</div></td><td><div class=\"dni-plaintext\">52.039684</div></td><td><div class=\"dni-plaintext\">-0.039684296</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "var actual = predictions.GetColumn<float>(\"fare_amount\");\n",
        "var predicted = predictions.GetColumn<float>(\"Score\");\n",
        "\n",
        "var compare = \n",
        "\tactual\n",
        "\t\t.Zip(predicted,(actual,pred) => new {Actual=actual, Predicted=pred, Difference=actual-pred})\n",
        "\t\t.Take(5);\n",
        "\n",
        "compare"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Just by quickly comparing the first few values you can see the predictions are generally just a few cents off from the actual amount. \n",
        "\n",
        "With ML.NET, you don't have to manually calculate the evaluation metrics for your models. ML.NET provides a built-in `Evaluate` method for each of the machine learning tasks it supports.  Use the `Evaluate` method for the regression task to calculate the evaluation metrics for the test set where the `fare_amount` column is the actual value and the `Score` column is the predicted value."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var evaluationMetrics = mlContext.Regression.Evaluate(predictions,\"fare_amount\", \"Score\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Using the `Evaluate` method not only calculates the metric you optimized for during training (R-Squared), but also all the metrics for the regression task. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th>MeanAbsoluteError</th><th>MeanSquaredError</th><th>RootMeanSquaredError</th><th>LossFunction</th><th>RSquared</th></tr></thead><tbody><tr><td><div class=\"dni-plaintext\">0.6107270253580241</div></td><td><div class=\"dni-plaintext\">6.673910566709432</div></td><td><div class=\"dni-plaintext\">2.58339129183123</div></td><td><div class=\"dni-plaintext\">6.673910534194763</div></td><td><div class=\"dni-plaintext\">0.9277130209892651</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluationMetrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Using R-Squared to evaluate the model\n",
        "\n",
        "Although you have multiple metrics to choose from, when you trained the model you optimized for R-Squared. R-squared (R2), or Coefficient of determination represents the predictive power of the model as a value between -inf and 1.00. 1.00 means there is a perfect fit, and the fit can be arbitrarily poor so the scores can be negative. A score of 0.00 means the model is guessing the expected value for the label. A negative R2 value indicates the fit does not follow the trend of the data and the model performs worse than random guessing. This is only possible with non-linear regression models or constrained linear regression. R2 measures how close the actual test data values are to the predicted values.\n",
        "\n",
        "\n",
        "For more information on other evaluation metrics, see the guide on how to [evaluate your ML.NET model with metrics](https://docs.microsoft.com/dotnet/machine-learning/resources/metrics)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train and evaluate models using cross-validation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### What is cross-validation?\n",
        "\n",
        "Cross-validation is a training and model evaluation technique that splits the data into several partitions and trains multiple models on these partitions. This technique improves the robustness of the model by holding out data from the training process. In addition to improving performance on unseen observations, in data-constrained environments it can be an effective tool for training models with a smaller dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Train a model using cross-validation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Start off by initializing the `MLContext`. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var cvMLContext = new MLContext();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, define your pipeline. In this case, the actual trainer is used instead of the `SweepableEstimator` from AutoML."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var cvMLPipeline = \n",
        "\tcvMLContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair(@\"vendor_id\", @\"vendor_id\"), new InputOutputColumnPair(@\"payment_type\", @\"payment_type\")},outputKind: OutputKind.Binary)\n",
        "\t\t.Append(cvMLContext.Transforms.ReplaceMissingValues(new[] { new InputOutputColumnPair(@\"rate_code\", @\"rate_code\"), new InputOutputColumnPair(@\"passenger_count\", @\"passenger_count\"), new InputOutputColumnPair(@\"trip_time_in_secs\", @\"trip_time_in_secs\"), new InputOutputColumnPair(@\"trip_distance\", @\"trip_distance\") }))\n",
        "        .Append(cvMLContext.Transforms.Concatenate(@\"Features\", new[] { @\"vendor_id\", @\"payment_type\", @\"rate_code\", @\"passenger_count\", @\"trip_time_in_secs\", @\"trip_distance\" }))\n",
        "\t\t.Append(cvMLContext.Regression.Trainers.FastForest(labelColumnName: \"fare_amount\"));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use the `CrossValidate` method to start the training and evaluation on your data using the defined pipeline. By default, data is split into five subsets but you can set this to any value you prefer using the `numberOfFolds` parameter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var cvResults = cvMLContext.Regression.CrossValidate(trainSet, cvMLPipeline, labelColumnName: \"fare_amount\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th><i>index</i></th><th>MeanAbsoluteError</th><th>MeanSquaredError</th><th>RootMeanSquaredError</th><th>LossFunction</th><th>RSquared</th></tr></thead><tbody><tr><td>0</td><td><div class=\"dni-plaintext\">1.3080861318665928</div></td><td><div class=\"dni-plaintext\">10.437064100433819</div></td><td><div class=\"dni-plaintext\">3.230644533283385</div></td><td><div class=\"dni-plaintext\">10.437063966978595</div></td><td><div class=\"dni-plaintext\">0.8884018879298109</div></td></tr><tr><td>1</td><td><div class=\"dni-plaintext\">1.360165666717988</div></td><td><div class=\"dni-plaintext\">10.798742155503342</div></td><td><div class=\"dni-plaintext\">3.2861439645127146</div></td><td><div class=\"dni-plaintext\">10.798742212339462</div></td><td><div class=\"dni-plaintext\">0.8850517190709868</div></td></tr><tr><td>2</td><td><div class=\"dni-plaintext\">1.2518280553341605</div></td><td><div class=\"dni-plaintext\">8.216770508755435</div></td><td><div class=\"dni-plaintext\">2.8664909748253935</div></td><td><div class=\"dni-plaintext\">8.216770540499184</div></td><td><div class=\"dni-plaintext\">0.9086191464533752</div></td></tr><tr><td>3</td><td><div class=\"dni-plaintext\">1.3420262660438258</div></td><td><div class=\"dni-plaintext\">10.229870802662239</div></td><td><div class=\"dni-plaintext\">3.1984169213319014</div></td><td><div class=\"dni-plaintext\">10.229870780300226</div></td><td><div class=\"dni-plaintext\">0.8886983637741152</div></td></tr><tr><td>4</td><td><div class=\"dni-plaintext\">1.5430782453416307</div></td><td><div class=\"dni-plaintext\">10.122911443078698</div></td><td><div class=\"dni-plaintext\">3.1816523133552317</div></td><td><div class=\"dni-plaintext\">10.122911478082717</div></td><td><div class=\"dni-plaintext\">0.88835065401792</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "cvResults.Select(x => x.Metrics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Calculate test set evaluation metrics\n",
        "\n",
        "Like in the previous example, use the `Evaluate` method on the full test set to evaluate the performance of the models trained using cross-validation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var cvTestEvalMetrics = \n",
        "\tcvResults\n",
        "\t\t.Select(fold => fold.Model.Transform(trainTestData.TestSet))\n",
        "\t\t.Select(predictions => cvMLContext.Regression.Evaluate(predictions, \"fare_amount\", \"Score\"));"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th><i>index</i></th><th>MeanAbsoluteError</th><th>MeanSquaredError</th><th>RootMeanSquaredError</th><th>LossFunction</th><th>RSquared</th></tr></thead><tbody><tr><td>0</td><td><div class=\"dni-plaintext\">1.2946164063640293</div></td><td><div class=\"dni-plaintext\">9.741100529118015</div></td><td><div class=\"dni-plaintext\">3.1210736180228134</div></td><td><div class=\"dni-plaintext\">9.741100436339362</div></td><td><div class=\"dni-plaintext\">0.8944318265394658</div></td></tr><tr><td>1</td><td><div class=\"dni-plaintext\">1.3517783571125017</div></td><td><div class=\"dni-plaintext\">10.697284288708751</div></td><td><div class=\"dni-plaintext\">3.270670311833455</div></td><td><div class=\"dni-plaintext\">10.697284293348838</div></td><td><div class=\"dni-plaintext\">0.8840692835505208</div></td></tr><tr><td>2</td><td><div class=\"dni-plaintext\">1.258738437439045</div></td><td><div class=\"dni-plaintext\">10.056973626481687</div></td><td><div class=\"dni-plaintext\">3.1712731869836897</div></td><td><div class=\"dni-plaintext\">10.056973680972417</div></td><td><div class=\"dni-plaintext\">0.8910085843878911</div></td></tr><tr><td>3</td><td><div class=\"dni-plaintext\">1.3321059289142654</div></td><td><div class=\"dni-plaintext\">10.119475327642087</div></td><td><div class=\"dni-plaintext\">3.1811122783771855</div></td><td><div class=\"dni-plaintext\">10.119475305281489</div></td><td><div class=\"dni-plaintext\">0.8903312286404623</div></td></tr><tr><td>4</td><td><div class=\"dni-plaintext\">1.5373861941022984</div></td><td><div class=\"dni-plaintext\">11.21522705311926</div></td><td><div class=\"dni-plaintext\">3.3489143096112897</div></td><td><div class=\"dni-plaintext\">11.215227048598013</div></td><td><div class=\"dni-plaintext\">0.878456132199452</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "cvTestEvalMetrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Model explainability\n",
        "\n",
        "Evaluation metrics are a good way to quantify how well your model makes predictions on new data. However, a good evaluation metric shouldn't be the only factor you consider when evaluating your model. To build more trust around your model and the decisions it makes, it's important to understand how and why it makes the decisions that it does. \n",
        "\n",
        "Models are becoming more commonplace in society and are affecting the lives of individuals. For example, let's imagine that a machine learning model was used to make a medical diagnosis. It's likely the model is right in its diagnosis but the stakes of making the wrong diagnosis are high due to the effects on an individual's health. Therefore, it's important for all stakeholders (patients, physicians, regulators) to understand what drove a model to make that diagnosis in order to feel confident in its decision. \n",
        "\n",
        "### Global and local explanations\n",
        "\n",
        "When explaining machine learning models, you can do so at the global and local level. \n",
        "\n",
        "Global explanations are when you make generalizations at the aggregate level. For example, let's say you're building a model to predict taxi fares. When explaining why some fares are more expensive than others, you'll most likely find that rides that rides that travel a longer distance or for a long period of time are likely to be more expensive. Although this doesn't tell you exactly why any one particular ride was more expensive than another, at an aggregate level you can see which features are important to the model when it makes its decisions. \n",
        "\n",
        "If you need more granularity when explaining your model's decisions, that's where local explanations come in. Local explanations allow you to see for any single prediction, which features contributed to the model's decision. For example, let's say a model is used to determine credit risk for personal loans. Given two customers with different amounts of debt, income, and payment history, the model determines which customer is more likely to pay the loan back. Using local explainability techniques, you are able to inspect at the individual level which features contributed to the decision of denying a loan. \n",
        "\n",
        "### Explanability techniques in ML.NET\n",
        "\n",
        "ML.NET provides two techniques for explaining models:\n",
        "\n",
        "- Permutation Feature Importance (PFI)\n",
        "- Feature Contribution Calculation (FCC)\n",
        "\n",
        "#### Permutation Feature Importance (PFI)\n",
        "\n",
        "Permutation feature importance is a **global** explainability technique. At a high-level it randomly shuffles data one feature at a time for the entire dataset and calculates how much the performance metric of interest decreases. The larger the change, the more important that feature is. For more information, see [Interpret model predictions using Permutation Feature Importance](https://docs.microsoft.com/dotnet/machine-learning/how-to-guides/explain-machine-learning-model-permutation-feature-importance-ml-net)\n",
        "\n",
        "#### Feature Contribution Calculation (FCC)\n",
        "\n",
        "Feature contribution calculation is a **local** explainability technique.  This technique computes a model-specific list of per-feature contributions to each of the predictions. These contributions can be positive (they make the evaluation metric higher) or negative (they make the evaluation metric lower)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Explain models with Permutation Feature Importance (PFI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Initialize MLContext"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiMLContext = new MLContext();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define data preparation pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiDataPipeline = \n",
        "\tpfiMLContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair(@\"vendor_id\", @\"vendor_id\"), new InputOutputColumnPair(@\"payment_type\", @\"payment_type\")},outputKind: OutputKind.Binary)\n",
        "\t\t.Append(pfiMLContext.Transforms.ReplaceMissingValues(new[] { new InputOutputColumnPair(@\"rate_code\", @\"rate_code\"), new InputOutputColumnPair(@\"passenger_count\", @\"passenger_count\"), new InputOutputColumnPair(@\"trip_time_in_secs\", @\"trip_time_in_secs\"), new InputOutputColumnPair(@\"trip_distance\", @\"trip_distance\") }))\n",
        "        .Append(pfiMLContext.Transforms.Concatenate(@\"Features\", new[] { @\"vendor_id\", @\"payment_type\", @\"rate_code\", @\"passenger_count\", @\"trip_time_in_secs\", @\"trip_distance\" }));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Apply data transformations to the training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiPreprocessedData = \n",
        "\tpfiDataPipeline\n",
        "\t\t.Fit(trainSet)\n",
        "\t\t.Transform(trainSet);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define your trainer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiTrainer = pfiMLContext.Regression.Trainers.FastForest(labelColumnName: \"fare_amount\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Fit the trainer to your preprocessed data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiModel = pfiTrainer.Fit(pfiPreprocessedData);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Calculate permutation feature importance (PFI)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var permutationFeatureImportance =\n",
        "    mlContext\n",
        "        .Regression\n",
        "        .PermutationFeatureImportance(pfiModel, pfiPreprocessedData, permutationCount:3, labelColumnName: \"fare_amount\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Extract R-Squared metric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pfiMetrics = \n",
        "\tpermutationFeatureImportance\n",
        "\t\t.Select((metric,idx) => new {idx, metric.RSquared})\n",
        "\t\t.OrderByDescending(x => Math.Abs(x.RSquared.Mean));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Get list of feature names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var featureContributionColumn = pfiPreprocessedData.Schema.GetColumnOrNull(\"Features\");\n",
        "var slotNames = new VBuffer<ReadOnlyMemory<char>>();\n",
        "featureContributionColumn.Value.GetSlotNames(ref slotNames);\n",
        "var slotNameValues = slotNames.DenseValues();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Map PFI metrics to feature names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th><i>index</i></th><th>Key</th><th>Value</th></tr></thead><tbody><tr><td>0</td><td>vendor_id.Bit2</td><td><div class=\"dni-plaintext\">-0.5103167076996584</div></td></tr><tr><td>1</td><td>vendor_id.Bit1</td><td><div class=\"dni-plaintext\">-0.20920573710951015</div></td></tr><tr><td>2</td><td>vendor_id.Bit0</td><td><div class=\"dni-plaintext\">-0.20524726245559932</div></td></tr><tr><td>3</td><td>payment_type.Bit3</td><td><div class=\"dni-plaintext\">-0.0013735609832817113</div></td></tr><tr><td>4</td><td>payment_type.Bit2</td><td><div class=\"dni-plaintext\">-0.0005371983420188927</div></td></tr><tr><td>5</td><td>payment_type.Bit1</td><td><div class=\"dni-plaintext\">-0.00015402329111213753</div></td></tr><tr><td>6</td><td>payment_type.Bit0</td><td><div class=\"dni-plaintext\">-7.256291485776185E-05</div></td></tr><tr><td>7</td><td>rate_code</td><td><div class=\"dni-plaintext\">-5.605438192501921E-07</div></td></tr><tr><td>8</td><td>passenger_count</td><td><div class=\"dni-plaintext\">0</div></td></tr><tr><td>9</td><td>trip_time_in_secs</td><td><div class=\"dni-plaintext\">0</div></td></tr><tr><td>10</td><td>trip_distance</td><td><div class=\"dni-plaintext\">0</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "var featureImportance = \n",
        "\tpfiMetrics\n",
        "\t\t.Zip(slotNameValues, (a,b) => new KeyValuePair<string,double>(b.ToString(),a.RSquared.Mean));\n",
        "\n",
        "featureImportance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Explain models with Feature Contribution Calculation (FCC)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Initialize MLContext"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var fccMLContext = new MLContext();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define data preparation pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var fccDataPipeline = \n",
        "\tfccMLContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair(@\"vendor_id\", @\"vendor_id\"), new InputOutputColumnPair(@\"payment_type\", @\"payment_type\")},outputKind: OutputKind.Binary)\n",
        "\t\t.Append(fccMLContext.Transforms.ReplaceMissingValues(new[] { new InputOutputColumnPair(@\"rate_code\", @\"rate_code\"), new InputOutputColumnPair(@\"passenger_count\", @\"passenger_count\"), new InputOutputColumnPair(@\"trip_time_in_secs\", @\"trip_time_in_secs\"), new InputOutputColumnPair(@\"trip_distance\", @\"trip_distance\") }))\n",
        "        .Append(fccMLContext.Transforms.Concatenate(@\"Features\", new[] { @\"vendor_id\", @\"payment_type\", @\"rate_code\", @\"passenger_count\", @\"trip_time_in_secs\", @\"trip_distance\" }));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Apply data transformations to the training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var fccPreprocessedData = \n",
        "\tfccDataPipeline\n",
        "\t\t.Fit(trainSet)\n",
        "\t\t.Transform(trainSet);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define your trainer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var fccTrainer = fccMLContext.Regression.Trainers.FastForest(labelColumnName: \"fare_amount\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Fit the trainer to your preprocessed data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var fccModel = fccTrainer.Fit(fccPreprocessedData);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Calculate feature contributions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var featureContributionCalc = \n",
        "\tfccMLContext.Transforms.CalculateFeatureContribution(fccModel,normalize:false)\n",
        "\t\t.Fit(fccPreprocessedData)\n",
        "\t\t.Transform(fccPreprocessedData);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Get list of feature names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var featureContributionColumn = featureContributionCalc.Schema.GetColumnOrNull(\"FeatureContributions\");\n",
        "var slotNames = new VBuffer<ReadOnlyMemory<char>>();\n",
        "featureContributionColumn.Value.GetSlotNames(ref slotNames);\n",
        "var slotNameValues = slotNames.DenseValues();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Get feature contribution values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var featureContributionValues = featureContributionCalc.GetColumn<float[]>(\"FeatureContributions\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Map feature contribution values with feature names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var featureContributions = \n",
        "\tfeatureContributionValues\n",
        "\t\t.Select(x => x.Zip(slotNameValues, (a,b) => new KeyValuePair<string,float>(b.ToString(),a)));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Display feature contributions for the first prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table><thead><tr><th><i>index</i></th><th>Key</th><th>Value</th></tr></thead><tbody><tr><td>0</td><td>vendor_id.Bit2</td><td><div class=\"dni-plaintext\">0</div></td></tr><tr><td>1</td><td>vendor_id.Bit1</td><td><div class=\"dni-plaintext\">0</div></td></tr><tr><td>2</td><td>vendor_id.Bit0</td><td><div class=\"dni-plaintext\">6.46816</div></td></tr><tr><td>3</td><td>payment_type.Bit3</td><td><div class=\"dni-plaintext\">0</div></td></tr><tr><td>4</td><td>payment_type.Bit2</td><td><div class=\"dni-plaintext\">-3.4585545</div></td></tr><tr><td>5</td><td>payment_type.Bit1</td><td><div class=\"dni-plaintext\">19.19871</div></td></tr><tr><td>6</td><td>payment_type.Bit0</td><td><div class=\"dni-plaintext\">10.600954</div></td></tr><tr><td>7</td><td>rate_code</td><td><div class=\"dni-plaintext\">-1779.6366</div></td></tr><tr><td>8</td><td>passenger_count</td><td><div class=\"dni-plaintext\">-1.6658905</div></td></tr><tr><td>9</td><td>trip_time_in_secs</td><td><div class=\"dni-plaintext\">75.95777</div></td></tr><tr><td>10</td><td>trip_distance</td><td><div class=\"dni-plaintext\">-803.3262</div></td></tr></tbody></table>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "featureContributions.First()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## How can I improve my model?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Model evaluation is an important step in the machine learning workflow to determine whether a model is ready to be deployed into production. If your model doesn't meet your criteria for being production-ready, there's a few things you can try to improve your model. These include:\n",
        "\n",
        "- **Reframe the problem** - Are you trying to solve the right problem? Consider looking at the problem from different points of view. \n",
        "- **Provide more data samples** - Experience is the best teacher. Providing more examples that represent your problem space help the trainers identify more edge cases. \n",
        "- **Add features (more context)** -  Building context around the data points helps algorithms as well as subject matter experts better make decisions. For example, the fact that a house has three bedrooms does not on its own give a good indication of its price. However, if you add context and now know that it is in a suburban neighborhood outside of a major metropolitan area where average age is 38, average household income is $80,000 and schools are in the top 20th percentile then the algorithm has more information to base its decisions on.\n",
        "- **Use meaningful data and features** - More data and features (context) can help improve accuracy but also introduce noise. Consider using permutation feature importance (PFI) and feature contribution calculation(FCC) to determine the features impacting your predictions and remove any features that don't contribute to your model.\n",
        "- **Use cross-validation** - Cross-validation can be an effective tool for training models with smaller datasets. \n",
        "- **Hyperparameter tuning** - Hyperparameters are just as important as the parameters learned during training. Finding the right ones is a process of trial and error. Use AutoML search spaces and sweeping estimators to help you find the right hyperparameters for your algorithm.\n",
        "- **Choose a different algorithm** - Just like hyperparamter tuning, finding the right algorithm to train your model is a process of trial and error. Use AutoML to help you iterate through the various algorithms available in ML.NET to help you choose the optimal algorithm to solve your problem. \n",
        "\n",
        "For more information, see the guide on [how to improve your ML.NET model](https://docs.microsoft.com/dotnet/machine-learning/resources/improve-machine-learning-model-ml-net)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Continue learning\n",
        "\n",
        "> [⏩ Next Module - AutoML Sweepable API](https://github.com/dotnet/csharp-notebooks/blob/main/machine-learning/05%20-%20AutoML%20Sweepable%20API.ipynb)  \n",
        "> [⏪ Last Module - Training and AutoML](https://github.com/dotnet/csharp-notebooks/blob/main/machine-learning/03-Training%20and%20AutoML.ipynb)\n",
        "\n",
        "### More End to End Examples\n",
        "\n",
        "- [Binary Classification with Titanic Dataset](./REF-Kaggle%20with%20Titanic%20Dataset.ipynb)  \n",
        "- [Value Prediction/Regression with Taxi Dataset](./E2E-Regression%20with%20Taxi%20Dataset.ipynb)  "
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".NET (C#)",
      "language": "C#",
      "name": ".net-csharp"
    },
    "language_info": {
      "file_extension": ".cs",
      "mimetype": "text/x-csharp",
      "name": "C#",
      "pygments_lexer": "csharp",
      "version": "8.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
